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This study describes^the probl ens. of searching' in professional 
journals for relevant articles on the subject of adult ecacation 
evaluation. - Seventyithree journals were identifiea as; containing 
§37 a^oprlate*; articles when^earching with eleyjn title words; 
43% of the titles referred £o human behavior change while 57-. 
referred to program development. Title^f unction worus,. who^ 
"what, " "why," "where, " ."when" and ,"how," were tabulateo. 
Methodology, practice arid issue words, were examined, as were .the 
words suggesting .evaluative , techniques and instr unentsv A 
micrl'thesaulus 1 suggesting ..method, technique and instrument was 
prepared! Search strategies using function words anc key .title 
worS were recommended to adult educators. Criteria for clearly 
titling journal articles were recommended to authors. 
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THE PROBLEM OF JOURNAL SEARCHING; 
THE CASE OF RETRIEVING ADULT EDUCATION EVALUATION ARTICLES. 

JOHN R A DOB SON . 
ST. FRANCIS XAVIER UNIVERSITY 

p ynbl em Statement 

Adult educators often have considerable difficulty identifying 
the articles relevant to their particular subject of interest. 
This is especially c true for the practitioner who does not search 
frequently in professional journals for articles of substance. 
The problem is not only one of knowing which journals contain 
articles' in' adult education, but also one of deciding what the 
contents of a particular ' ar ticle might°be on the basis of the 
words, in the title. The problem' of knowing the meaning- of words., 
frequently used by researchers and other professionals is 
. aggravated' by/ '-the, lack' of consistency in usage of adult education 
termino logy. This, is not a new problem for "educat or s; Goodman 
observed in 1972 that t 

Education does not now appear to be characterised by, 
.•a singular prof e'ssion, a singular discipline/ or 
.., ' monolithic institutions. Pluralir-m is the order of 
the "day. - Professional organizations proliferate; , 
adherents, of one discipline after another appear and '* 
argue the advantages of a particular approach /.the 
attractiveness of . .a ' different conceptual framework;- >. 
institutions which. untiL recently seemed quite, 
vt net able k are attacked from all .sides. (Goodman, 
1972) - ■ - /• • f,":: ' ; /: 



The words of a title to an adult education article are ..the, .keys 
provided by the author to prospective- readers ... suranari zing the 
ideas and experiences contained 'in the body of the paper. The 
title serves as a primary' retrieval system ^whi^h allows for 
information collected and articulated in writing by one personate 
be'; retrieved- by another. Descriptor terms are the keys created 
by • document cataloguers in order to provide for an alternative 
retrieval /system .in the ' information gathering , process. The 
current state of the art of the profession of adult educatrbn is 
reflected in the vocabulary created by both authors and 
."cataloguers^ 

There are basically two approaches to information retrieval: 
through the examination of title words, and through°.the searching 
for concepts in. appropriate thesauri of descriptor terms. Each 

approach: ,. . attempt s , to provide : searchers ' with -staff i cicnt 

information to permit a decision to be made, as to the relevance 
of the particular document. / A e disadvantage, /in title- word- 
searching is that the key words provided are either insufficient 
to make a judgement or are so creatively innovative as to be 
.inconclusive. On the other hand, a disadvantage, in descriptor 
searching is that key terms usee attempt to categorize a 
particular article within, the. conventions set down by an 
institutional authority. This often does .not allow, for the 
dynamic developments taking place within the field of inquiry. 
Again, Goodman. observed: 

A great many terms are necessary to describe the many 
aspects " of education, and the task of relating them. 



in even "an approximately : -consistent ■ *way fc is_ : an 
1: enormous one * The undertaking ; should be .managed by . 
people who not only know T . what' they are talking, about, 
but who also shquld v be° able to predict what" people in 
their fie^d are/ likely to be talking -abqut in the 
nbar £utur£,v It . should also ..; enlist people who ..are " 
willing to pay a great deal of attention to. the - 
details of relating one term to another within the 
sy.ste.nu (Goodman , 1972) 
A * second disadvantage in descriptor searching is that; the 
practitioner unlike the researcher/ is generally unfamiliar v/ith* 
the several thesauri. ^available in -reference collections. \ 

An analysis of title words and descriptor terms found in recent 
adult education- evaluation articles will/assist practitioners in 
their guest ^f or information. \% f . \ 

Litera tu re •' ■ ,* ~ 

The basic references for -adult educators seeking relevant 
professional vocabulary are: ~~ 

The Ency clopedia of Educati "ori 

Thf?n ai1r WH nf EPTC 'Descriptors 1 

Ifoefiaurug of DeveloyTnent Corm an i cations 

QEgD Mag fQ^hegatirtigt EgQnQTnig and Social Development 

MSP Wan^fQftnal Education The saurus ; 

In addition- to these thfre atV several subject heading^indices 
based upon the Library of Congress or Dewey cataloguing system. 



These thesauri build their literature, search strategies- on 
artificially controlled words categorised as broad (DT) , narrow 
(NT) and related (RT). 0 terns/ as.. well as alphabetized rotated 
listings. 

The' OECD r.' acrbthasaur'iis provides a guide to broad and. narrow 
terns used in the social sciences/ including education/ but due 
to 'the-. ''wide scope of its topics does not. allow . ; t : or- nuch 
'specificity in searching, for a subject as narrow v as "adult 
education evaluation." Indeed/ the only descriptive .-rfcr row. terras 
that were useful in . -searching -were "methods r w "techniques./ " 
"evaluation/" "research/" and "analysis"- /when, cross-ref erericed 
with, "adult education." • /;> ,f ■ 

When using the ERIC Thesaurus^ a wider / variety, of' narrdw terfus 

v/ere available as suggested ''search- wjbr&s. These included . 23- 

. , • . .. - -. - : .v. .\- • . •, \ ' -v :,, 

techniques of analysis^ 3 of evaluation^ 3 of assessments 2 of 
research and 4 of; surveying. (See : Table 0 7 .) 

All the* thesauri reviewed relied "heavily "on the use • of a 
non-educational subject descriptor to reduce the search field. 

While there"! is Considerable literature dealing with the structure 
and procedures used In creating 'thesadri bf descriptors "very 
little^ is, known about' the people who* have bee ; n using the ERIC 
Thesaurus, why and how they Tiave been using ity ajid what they 
think of. it." '(Goodman/ 1972) - 



During the past several months students in the Department of 
Adult Education at St. Francis; Xavier University have been 
searching for articles relevant to the profession. Close to 3000 
articles were) identified in 73 journals published in 1981 and 
1982. The articles were catalogued on an HP3000-300 computer 
using . the MINIS IS information management system; The inverted 
file capability of the o system allows for both title and 
descriptor search strategies. 

The Department's database is used by students, researchers and 
practitioners alike as the source of written information on the 
theory and practice of adult education. Students with sufficient 
available time, and researchers with considerable knowledge have 
little difficulty in retrieving information on their' particular 
topic. Practitioners, . on the other hand, with less time- to 
search and virtually no knowledge of the variety of synonomous 
terms have considerable difficulty in. accessing, the literature. 
Goodman has noted that: 

One of the major questions to be answered about 

t 

* ■ •* •" 

Thesaurus' use is the extent to which it may be 
utilized to. aid' the kind, of search performed by a 
. professional searcher acting as an intermediary 
..: between -.the actual, user, and the, system. It may. be 
that a great many future searches of the system will _ 
involve, "an. intermediary, person. This possibility 
should not be viewed simply as an aid to the user; it 
should also, be viewed as, a .potentially 'inevitable 
barrier between the uraer and the system. , In the \ 



first instance one can contemplate a highly valuable 
interface between a il ^^ e us ^ r .> ani 4tha couple:: 
■ system , an interface in the form .of a searcher wbo 
translates- between - .the user's' natural language ana 
the system's artificially controlled language. 
(Goodman, 1272) - / . 

An alternative to the growing mystification of . the professional 
language of adult education is the development of easy to use 
search strategies that can .be applied to the words of ° the title 
in -such a way as to provide ready access to relevant information. 
Adult education h§s traditionally been characterised by its 
practical approach to learning, consequently any effort to 
demystify die profession is to be valued^ 

Objectives of ■ Study . ' 

This <* study examines the problem of determining the 
appropriateness, of title words- as guides to the selection of 
articles written on the topic of adult education, evaluation. It 
suggests to practitioners- ways, of approaching the problem of 
literature search and to authors ways of titling their work so as 
to make it readily accessible. 

The objectives of. this study are: 

1. To identify relevant adult education journals with articles 
on evaluation? 

2§ To identify relevant evaluation words; 



3. To categorize words of a title according to their function; 

" • 

4. To analyze evaluation method, techriique and instrument words; 

5 j to suggest appropriate title search strategy 'for - the topic of 
evaluation; and •. 

6.», To suggest appropriate title writing criteria. 

ppf-Tnifcions , . ■ \ • •- 

i_ Evaluation: 'the determining, or setting, of value; 

2. Function: the purpose for which s omethi n g is designed or 
exits ; 

3. Instrument: a device for measuring the present value. of the 
quantity under observation [synonym: tool] ; 

4. Ilethod: a plan" or. system of action, inquiry, analysis, etc. 

5. Research.: systematic inquiry into .a subject in order to 
discover or revise facts, theories, etc.; 

6. Search: to explore or examine in order to discover; 

7. Technique: the body of specialized procedures used in any 
specific field. 

The methodology used to identify, relevant articles and words, was 
' developmental. .'" ° 



1. Journals in the social sciences were searched to render 
articles in £dult education; 

& » * 

\ . ■ ■ ■ . • 

2* Acluster of words' suggesting that the article was relateo to 
evaluation were identified; 

3. " Article abstracts were reviewed to determine suitability; 

4. Evaluation title words were categorised as to function:- 
"what," "who," "why," "where," "when" and "how; n 

c- 

5. "Why" evaluation words were categorize as guides to 
metho dolog y/ pr acti ce and current issues ; 

6. "Kow" evaluation words were ^categorized as to methods, 

techniques, . and instruments; 

• . .. _j — .. 

-7. A microthesaurus of evaluation related words was preparea. 
Pi rul-i nqs "find- Discuss ion 

The .• findings have been placed in tables at the end of the paper: 
The following discussion attempts to highlight certain features 
of the data. 

1. Jgpnals with articles in 'adult education evaluation. (See 
Table 1.) --• 

Seventy three journals' containing 2977 adult education articles 
published in 1981-82 were searched; . 51 contained 324 evaluation 
articles; DEVSIS contained 101; ERIC was unavailable for 
searching on the informational retrieval system at the time of 



'thfesfcudy". Journals with the largest number of "hits" on 
evaluation were those-' related either to community development or 
to human resources training.: Journals in general education and. 
in the social .. sciences containing, adult education evaluation 
articles, were few. 

2. * Evaluation words, used to suggest method. (See Table 2.) 

Eleven words were considered to be synonymous with evaluation; 5 
words were most frequently (SS%) used to suggest that the article 
was referring to evaluation; the- 6 other words appeared less 
frequently' (125). Although the' word "research" appearea in 25% 
of the titles -it was redundant, with another evaluation word in 
-20% of the references. The distribution of title wordc between 
the ."analysis" of human performance (435) and the "evaluation" of 
programs (575) was a very interesting finding. 



3. Function words in titles. (See Table 3.5 

"Uhat" kind of evaluation took place and "how" it was conducted 
appeared frequently, "who" and ; ? where" less' frequently , while 
"v/hy" and "when" were seldom manifested in ? the title. 

•** • * ■ 

'Using the "why" function to mean "why write a ■'report" made It a 
useful approach for . identifying ' (77%) guides to methodology, 
practice and current^ issues. ■ * v 

The use of "who" and "where" words, • although limited (38%), 
provided a context, to the article in question. The designation 
of people of a class or type ("who") clearly limited tne stuay uo 



-a target .population which was readily defineable. Title words. > 
designating .."where" the evaluation too!; place orten was a 
substitute to "who" serving the sane purpose of targeting the 
population being evaluated. 

"Uhen" the evaluation cook place was, generally absent iron the 
title- • v ' 

4. Title words suggesting methodology, practice and issues* (See. 
Table 4.) 

Title' words often, .but not always <77£) r <3*ve the 'reason why the 
article .was written- 

■ Uords suggesting that the article focuses' on Iiq<; to conduct an 
evaluation were! varied. Twonty two different worc^ poirrtec to- 
articles on methodology (44%) v Uords such ^ as "approach/ " 
"strategy" and "system" . referred- to general articles on methods 
while words such as "techniques," "levels," "st-pc" ar.d "tactics" 
usually referred to the specif ics ...of evaluation procedure*: ^ t 

The practice of evaluation is generally referred , to in the title 
(20%) when such v/ords as "report", "project", and "program" ■ 
appear. There are only- a- few evaluation reports 'found in 
journals. The DEVSIS and ERIC information systems / are more 
frequently the repository of reports., 

Current issues in evaluation were infrequently (135) suggested by 
words in the title. 'Words such as "roles"/: ■-. "problems"', 



^concepts", "policies" - and "issues", pointed to. articles which 



discussed the current state T or evaluation.- 



5. Title words suggesting quantitative, .^qualitative, 
participatory/ formative and summative^ evaluation technique^. 
(See Table 5°.) . . . " 

"¥^Tua"tri~oTr — tl-tl-es-^^er^_£eq^re^ to include one of the eleven 
method words in order to - be ; considered in thl^^stud^fv^In 
addition to one of these general words the titles also included a 
second ^ word which suggested ^ particular approach, :,or technique, 
being used in evaluation. Tha woi;ds describing technique divided' 

-into .f. "quantitative, " "qualitative, " "participatory, " "formative" 
and ""summative" evaluation | approaches.- " There v/aa, nowever, 
considerable • overlapping in approach between qualitative 
participatory as well as between formative and .summative 



6 ^ * Title words suggesting evaluation . instruiaents. : . ( See Tabic 

6.) . • ; , r V '-\ 



An evaluation title: of ten - had- an alternative sec^nd^ word 
ingesting aiC instrument i (36%) ^ Words ' such as/ "surrey" and 
"test" \ \were used so frequently that; , they of ten stood alone iiT a 
ti tie r to sugg e st a method, >xri; themselves. , '^yy/'-y'.; . 

-7. A micro-thesaurus, of evaluation related words; ;; (S3e Table 

\ • .•>■•" •"; ? ■ ■ \ ' s*y>; . ' 



One hundr etr """and forty-one words were identrfied-rfrom titles 
and/or thesa uri that suggested some aspect — method, technique 
or instrument of evaluation. 6f ; th, totaj, 29 were ttirrns 

drawn from <-hr> Thasaur . of ERIC ^ Rgriytor^. Only two of the 
ERIC terms were found in ,ae titles searched I 



Conclusions . , - — 

1. Apart from those journals whose purpose it^is to disseminate;; 
e^^uatTM~Tr"esearch there 7 are numerous other ^serials v;hich 
; publish evaluation information occasionally * Collectiveily # these 
I*££ter journals off er J the reader half of the total 'num^r^of ; 
ref er<i^es on the subject • • . ' -""^ ;> ; 7- : 



The Inter natioh^^evelopment* Res^rch, Centre f s DEVSIS collection 
and the^ ERIC Cl^S^^iouae on- .Adult, ;: Career ;, and Vocational 
Educatior^>rpvid^ resources '.}. for reports on the 

== pr^txcar = iss :u^s^ii<£^ethH0cls 



2. ; Identification of ^appropriate 
evaluation words in a title 
searching \f or information. - 




rticles Sy usiny ; key • 
Ltagy to follow v/hen 



V3V The selection., of function words in .a title is a^usefiil 
■ •' strategy:;^ -f ok.' identifying the "What •••,,;< . "how",- and n wh6/vmere^:o| 
>^the .evaluation. -Xt- ia also, useful. . in. detecmining^the^purposer 
"Why""-, of t^e article. ; It wl^ nol: useful In identifying the 
- "when 1 ? of an. evaluation. jf> v S'-'-\ 



.4. A thesaurus of descriptors is a helpful support when 

searching: foe articles which have .been incompletely or 

inappropriately titled/, however # it is not as useful as the 

solitary source of searching words • * 

Recommendations' \ . - 

The recommendations, resulting ' from this study are ' diracte'd 
primarily to the practitioner ^who is searching for relevant; 
articles and to authors who want : to make their work. more, 
accessible* />/ _ 

- 1. The adult educator seeking to identify appropriate eval lotion 
articles; without the aid of computer searching facilities would 
do . best to review journals in community development/ adult 
education, and., human: resources training, ^d..:Vuse;^e - nine title 
key evaluation words as a guide to selection* 

~^2^^Ther^u^^ searching f acil i t ies : should 

V e*>t<2r the activity^Mth^ 

selection / be comes ; iri : itself # a v 
{The first selection of titles will prbbabiy be inadequate/ and 
• successive v passes at: the database will gradually 
jlnf ormatiori bei«g sought. 

terms that, seem appropriate to ^he topic and try to identify 300 
: ^ ; /' to r: '|^ The minirdatabase so; created shbuld then be 

-cross-ref erenbec^a^ to the .guide: words suggested for each 



should, render between 10 and 20 hits each; these then can b ; e 
perused.. WherTperusincpa set of "hits" the secondary adjective 
words which describe/ or modify, .the primary, evaluation words 
should be the focus of the search. Eventually , the articles 
appropriates to the interest of the adult educator will emerge* 
One hour is not an uncommon length of time to spend searching^f or 
15 to 30 articles of. interest. .* ^ 

4 . The author sv of journal articles should consider that the 
7tI£Ie^ *^'%he ptimsLtY:; resources that an adult educatgr has for 

retrieving the contents of their work. Although clever or 
creative titles are attractive, to an ■•in. group" of readers who 
are current knowledge of the trends and authors* found in 
particular/ journals/ such titles generally* exclude, the' casual 
reader . ^ Jhe title should be written in plain English. -,y : " 

5. Authors should include a combinatioii.of words in the title 
which tell the prospective reader what is; being evaluated/ how it. 
is-being evaluated and why it is being evaluated. .* Other function 
words should be included when the target; population, or. the 
location , or the time are the principle subject being evaluated. 
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TABLE ONE: . : 

JOURNALS WITH ARTICLES III ADULT EDO CAT IOH EVALUATION 

* . 1981 - 1982 

• / Adult 

■ • Ed. 



ADULT E DUCAT 1017 
ADULT EDUCATION 
ADULT EDUCATION 
ADULT EDUCATION 
ADULT. EDUCATION 
ADULT EDUCATION 
ALBERTA JOURNAL 
AMERICAN JOURNAL 



(GREAT BRITAIN) 
(UNESCO)- ° 
(UNITED STATES )' 
AND DEVELOPMENT - -. 

IN FINLAND 

OF EDUCATIONAL RESEARCH 
OF SOCIOLOGY 



AUSTRALIAN JOURNAL -OF ADULT EDUCATION 

BRITISH JOURNAL :0F EDUCATIONAL PSYCHOLOGY 

BRITISH JOURNAL OF EDUCATIONAL 
BRITISH JOURNAL OF EDUCATIONAL 
CANADIAN ADMINISTRATOR < 
COUNSELLOR . 
JOURNAL OF EDUCATION- 
JOURNAL' OF HIGHER EDUCA" 
JOURNAL OF SOCIOLOGY 
MAGAZ INE OF HIGHER LEARNING 
HOUSE 
COLLEGE REVIEW .. 
DEVELOPMENT JOURNAL 



CANADIAN. 
CANADIAN' 
CANADIAN 
CANADIAN 
CHANGE: 
CLEARING 
COMMUNITY 
COMMUNITY 



STUDIES 
TECHNOLOGY 



HON 



ERIC 



COMPARATIVE EDUCATION REVIEW 
. CONTACT 

CONVERGENCE . 
- COUNSELLING AND HUMAN DEVELOPMENT 

-Cmr-Tafit-OTi^lr-E&EtffiilT--^-^— — ' . • -. > ■ • ■ ' : 

DELTA PI EPSILON 

DEVELOPMENT 'DIALOGUE 

DEVELOPMENT DIGEST - 

ECONOMIC DEVELOPMENT AND CULTURE CHANGE 

EDUCATION AND TRAINING OF MENTALLY RETARDED 

EDUCATIONAL COMMUNICATION AND TECHNOLOGY 

EDUCATIONAL RESEARCH 

EDUCATIONAL TECHNOLOGY 

•, educational theory; 
evaluation: u. ' s.* rev ie1t , 
Group and organizational studies 

HARVARD: EDUCATIONAL REVIElf 
HUMAN RELATIONS , (UNITED STATES) . 
INDIAN JOURNAL. OF. ADULT EDUCATION 
INFOS YSTEMS . . 

INTERNATIONAL DEVELOPMENT REVIEW 

iilTERHAT IONAL JOURNAL OF .INTER CULTURAL RELATIONS 
INTERNATIONAL REVIEW OF EDUCATION 
. JOURNAL ^OF, CURRICULUM STUDIES " ::r'r 
JOURNAL. OF CURRICULUM THEQRIZ ING ' ' 
JOURNAL OF EDUCATION t : 
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9 
2 



1 

0 ; 

0 . 
0 

1 . 

,'.0-\ 

l 

f, 

o 

: 2 

'24 

3 

3 
12 

1 
_1_1 

1 

7 
13 - 
19- 

,0 
20 

io 

20 

3 

7 
12 

3 
12. 

0 

0 
16 
15 - 

1 

1 

X 



JOURNAL 



OF 
OP 
OP 
OP 
OP 
OF 
OF 
OF 
OF 
OF 



EDUCATIONAL MEASUREMENT 
EDU CAT ION AL RES EARCH 
EDUCATIONAL THOUGHT 
EXPERIMENTAL EDUCATION 



EXTENSION 

HUMANISTIC PSYCHOLOGY 

READING • . ' . 

SOCIAL ISSUES' . • • • 

SPECIAL EDUCATION ... 

THE COMMUNITY DEVELOPMENT SOCIETY 



JOURNAL 
JOURNAL 
JOURNAL 
JOURNAL 
JOURNAL 
JOURNAL 
JOURNAL 
JOURNAL 
JOURNAL 

LIFELONG LEARNING (ADULT LEADERSHIP) : 
HCGILL JOURNAL' OF EDUCATION 
MEASUREMENT AND EVALUATION IN, GUIDANCE 
PEAEODY JOURNAL OF EDUCATION 
PERSONNEL AND GUIDANCE JOURNAL 
PHI DELTA KAPPAH 

PROGRAMMED LEARNING AND EDUCATIONAL TECHNOLOGY 
RESEARCH m HIGHER EDUCATION : 
REVIEW OF EDUCATIONAL - RESEARCH 
SOCIAL ACTION 
TEACHERS COLLEGE RECORD , 
TEACHING AT A DISTANCE 
THEORY INTO PRACTICE 
THIRD WORLD QUARTERLY. 
TRAINING AND DEVELOPMENT JOURNAL 
- TRAINING i HUMAN -RESOURCES DEVELOPMENT 



6 


r. 


19 


5 


•> 


0 


4 


0 


10 


c 


48 


3 


7 


2 


1 


3 


*> 


i 


10 


2 


55 


3 


l 
2 


0 


" 3 


3 


5 


3 


11 


0 


8 


0 


99 


15 


5 - 


•o 


7 


4 


26 5 • 


3 2 




0 


D . 


1 


o ■ 


0 


495 


32 


301" 


32 


35 





TABLE TWO: 
EVALUATION WORDS SUGGESTING METHOD 
CROSS-REFERENCED WITH HUMAN . BEHAVIOR AND PROGRAMMING 



• 


HUMAN 


PRO 


G RAM 


TOTAL 


% 




BEHAVIOR 


PRE- 


POST- . 






JUDGEMENT - 


4* . 


0 


o 


4 


+ 


APPRAISAL 


5 


0 


1 


6 


+ 


MEASUREMENT 


6 


o 


2 


8 


+ 


OBSERVATION 


8 


0 


0 


a . 


+ 


SURVEY 


■• 2 


10 


0 


12 


+ 


TEST 


13 




•7 

7 




1 Oft 


ASSESSMENT 


24 


20 


12 


• 56 


9% 


EVALUATION 


20 




53 


101 


17% - 


CAS 3 STUDY 


- 27 


0 


80 


107 


| 1 8% 


ANALYSIS 


77 


17 


20 


114- 


! 19% 

j • 


RESEARCH 


75 


20 


54 


149 ' 


! 25% 




(261) 


(104) 


( 229) 


(594) J 




43% 


57% 


j< 100%) _ 

i 




TABLE THREE: 
FUNCTION WORDS IN TITLES 



"Who" was being evaluated: 
People of a type or class 

"What" was being, evaluated: 
Human behavior 
Program planning 
Program outcome 



What learning content was being evaluated; 
ABE, literacy, life skills 
Vocational., economic development 
Social welfare, human services 
Health, nutrition, fitness 
Culture, art, folklore, religion 



"Why" report oh evaluation: 
Guides to method 
Current issues 
Practice 



"When" the evaluation took place 



20% 



43% 
19% 
3 8% 
(100%) 



2% 
6% 
2% 
2% 
1% 
(13%) 



44% 
13% 
20% 
(77%) 

3% 



"Where" the evaluation took place 
In a learning environment 

/Informal 2% 
Non-formal 3% 
Skill training 4% 
Formal . 1% 
In an organization ■-; 
In a community 
In a geographic locale 



10% 



8% 
8% 

< 2% 
(28%) 



"How" was the evaluation carried out 
Method " 



Technique 
Instrument 



100% 
3 4% 
36% 
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TABLE FOUR: 

GUIDE WORDS SUGGESTING METHODOLOGY, PRACTICE AND ISSUES. 



Methodology 












CiLiCtX'LDiN ID 




DECISION 


T~ ■"' ' ' 

7 


MODELS 


18 


KIT 


2 


DESIGN 


7 


DESIGN 


22 


SCHEME 


2 


FRAMEWORK 


8 


" GUIDE 


27 


PROCEDURES 


3 


HANDBOOK 


10 


PRACTICE 


27 


TACTICS 


4 


FORM 


11 


SYSTEM 


28 


STEPS 


5 


HOW 


12 


TECHNIQUES 


33 


LEVELS 


6 


PROCESS 


14 


STRATEGY 


36 1 




j 




p 


APPROACH 


40 ! 

1 


Practice 










i 

1 


OUTCOME 


; ' 2 ! 


SUMMARY 


10 


PROJECT 


47 


PRODUCT 


: 9 

i 
i 


REPORT 


30 

t 


PROGRAM 


105 . 

I 


Issues\ 












VIEWPOINT x 


1 


CHALLENGES 


3 . 


ISSUES 


; 10 


PRIORITIES 


X 2 


PRINCIPLES 


4 


POLICIES " 


12 


PROFILE 


■ 2 


TRENDS 


5 


CONCEPTS 


13 


TERMS 




DEFINITIONS 


9. 


PROBLEMS 


17 






*x. 




ROLES 


j 20 



♦Frequency count. 
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TABLE FIVE: 

WORDS SUGGESTING QUANTITATIVE/ QUALITATIVE, PARTICIPATORY, 
- FORMATIVE AND SUMMATIVE TECHNIQUES. . 



Qualitative Techniques 



! BASIC 
; CENSUS 



COMPETENCY 
COST— 

CRITERION- 
DIFFERENTIAL 
EFFICIENT 
EMPIRICAL 



EXPERIMENTAL 

FACTOR 

ITEM 

LONGITUDINAL 
NOMINAL 
OBJECTIVE 
OPTIMAL 
PATTERN c 



POOLED. 
PURE 

QUANTITATIVE 

RATING 

RESPONSE 

SAMPLE 

SYSTEMATIC 

TREATMENT 



Qualitative Techniq ue 



ATTITUDE 
COMPARATIVE 
CONDITIONAL 
CONTEXT 



EFFECTIVE 
EXPERIMENTAL 
FIRST-HAND 
MULTI-DIMENSIONAL 



MULTI-LEVEL 

OPTIMAL 

PRACTICAL 

QUALITATIVE 

SIMULATION 



Participatory Technique, 



COMMUNITY 
CUSTOMIZED — --" 
DELPHI 


GROUP 

INTERACTION 
INTERVENTION 


PARTICIPATORY 1 
SELF- 
TRANSACT IONAL ! 

! 

- ■ J 


Formative Technique 






ANTECEDENT 

ANTICIPATORY . 

BACKGROUND 

CONTINGENCY 

CONTINUING 

DAILY 

DEVELOPMENTAL 


ESTIMATION 

FRONT-END. 

GIFT -HORSE 

INTRODUCTORY 

MONITOR / 

NEEDS 

PERFORMANCE 


PLANNING/--^ 
PREPARATIVE 
PROBI1 
^PROBLEM 
^SKILLS 

TRANSITIONAL 

■ 


Summative Technique 


ACHIEVEMENT 
COGNITIVE 
• CONTENT 

INSTRUCTIONAL 


KNOWLEDGE 
OUTCOME 
PRODUCT 
PROGRAM 


PROJECT 
RESULTS- 
SUCCESS 
TRAINING 



TABLE SIX: 

TITLE WORDS SUGGESTING EVALUATION INSTRUMENTS 



FORM SCALE 
INSTRUMENT 1 SURVEY 

QUESTIONAIRE TEST 

v 

RELIABILITY TOOL 




------ j 

TAXONOMY 

TYPOLOGY 

UNOBTRUSIVE 

VALIDITY 



o 




MICR0TI1 
METHOD , 



, * ■ TABLE SEVEN : 
ESAURUS SUGGESTING EVALUATION 

technique or. instrument 



§ 14 
* 

114 * 
2 

+ 

6 * 

56 



107 
5 

. * 

6 
* . 
* 

. * 

' 17 
22 
■ * 



ACHIEVE! lEMT 
ACTION . * 
ANALYSIS 
ANTECEDENT 
ANTICIPATORY 
APPRAISAL 
ASSESSMENT . 
ATTITUDE 
BACKGROUND 
BAS IC 

CASE STUDY 
CENSUS 
. CLUSTER. " 
COGNITIVE 
COHORT 
COMMUNITY 
COMPARATIVE 
COMPARATIVE 
COMPETENCE 
COMPONENTIAL 
CONDITIONAL 
CONTENT 
CONTEXT 
CONTINGENCY 
CONTINUING " 
COST- 
COST-EENEFIT 
COST-EFFECTIVE 
COST-EFFICIENT 

COVARIAl!CE : ' * 
CRITERION-REFERENCE 

CROSS— 3 
CRCS S -CULTURAL 
CROSS -GENDER „. 
CROSS -SECTION 
CUSTOMIZED 
DAILY' 
. DATA * 
DELPHI 

DEVELOPMENTAL 
DIFFERENTIAL 
DIRECT 

DISCRIMINANT 
DOUBLE.-BLIND 
EDUCATIONAL 
EFFECTIVE 



15 



-17 
13 



33 



.10 " 

6 
* 

13 



EFFICIENCY 
EMPIRICAL 
ERROR 
ESTIMATION 
EVALUATION 101 *":■ 
EXPERIENTIAL , 5 
EXPERIMENTAL 10 
FACTOR 6, *- 

FIRST-HAND . 
FORM .12 
FORMATIVE * 
FRONT-END 
GIFT-HORSE 
GROUP 

INFORMAL . • *. 

INPUT * 
INSTRUCTIONAL . 
INSTRUMENT 8 
INTERACTION •. ,= " * 
INTERCUL TUR AL , 
INTERVENTION 
INTRODUCTORY 
ITEM ■ 1 * 
JUDGEMENT 4 
KNOULEDGE 
LONGITUDINAL ' 
MAIL . 

MEASURE 8 
' MEMORY ' • . 
MODEL f 
MONITOR 4 . 

MOT7VAT IONAL - 
MULTIDIMENSIONAL 
MULTILEVEL > 
NEEDS 67 * . 

NOMINAL -GROUP : " 
NON-VERBAL' ■ .., 
OBJECTIVE -EASED 
OBSERVATION 0 
OCCUPATIONAL- * 
OPTIMAL 

OUTCOME 2 
OUTPUT * 
PARTICIPATORY 38 
PATTERN 

PERFORMANCE 13 



PLANNING- 
POOLED 

PRACTICAL .' 
PREPAEAT IVE 
PROBE 
PROBLEM 
PRODUCT 
PROFILE 
PROGRAM , 
PROJECT 
PURE 

QUALITAT IVE 
QUANTITATIVE- 
QUESTION- . ' 
RATING 
REGRESSION ' 
EELL\B ILITY 
' RESEARCH -149 
RESPONSE 
RESULTS 
iiL VIEU ' 
ROLE 
SAMPLE' 
SCALE 
SELP- o 16 
SIMULATION 
SOCIAL 
SPATIAL 
STATISTICAL ' 
STUDY ,,: '- 
SUCCESS 
SU Mi CAT IVE 

survmy „/ 

SYSTEMATIC ■ 
SYSTEMS .' 

TASK .•/ ... 

TAi:ONCilY ; v ■ 
■ TEST 
TOOL 

/TRANSACTIONAL 
' TRANSIT ION AL 
' TREATMENT 
TREND - 
TYPOLOGY 
UNOBTRUSIVE 
VALIDITY . : 
VARIANCE 



* 

10 5 

• 47 

• 3 

28 

"2 
* 



14' 



33 



12- 



2 ! 



2 



* Items listed in the Thfflmifnn Q£ fiftT r ^scrintors, 

# number of- hits-,reaii 
+ Items where only one 



.zed during the\evaluati 
**hit was realized. / 



search. 



